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Costly Progress: De Blasio’s Renewal School Program 


Executive Summary 


n November 2014, the New York City Department of Education (DOE) classified 94 

low-performing schools as “renewal schools.” New York’s Renewal School Program 

(RSP), announced Mayor Bill de Blasio at the program’s launch, represented a big 
change to the city’s education policy: whereas former mayor Michael Bloomberg favored 
closing failing schools, RSP would fix them instead. 


RSP—which went into full effect in the spring of 2015—has retained many of the core elements of a strong ac- 
countability policy, including school oversight and more time in the classroom (renewal schools have longer 
days). RSP supplements these stalwarts of education reform with an infusion of extra money: $149 million over 
three years. 


Is RSP succeeding? The release of renewal schools’ spring 2016 state test results in math and English language 
arts (ELA) allows for a one-year evaluation. This report applies regression models to measure how renewal 
schools perform relative to how they would have fared absent RSP. 


Key Findings: 
~) RSP has boosted student achievement by a meaningful magnitude, on average. 


~) When the demographic profiles of their students are accounted for, about 33.38% of renewal schools still perform 


worse than expected in ELA, and 38.1% perform worse than expected in math. 


~) RSP is also significantly more costly than Mayor Bloomberg’s school-closure policy. 
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Introduction 


n November 2014, Department of Education chancellor Carmen Farina 

designated 94 of the city’s worst schools as “renewal schools,” targeting 

them for substantial turnaround efforts. Some received extra support 
almost immediately; at others, implementation of RSP was delayed.’ By the 
spring of 2015, the mayor’s Renewal School Program (see sidebar) entered 
into full effect in all 94 renewal schools. 


The DOE initially attached a three-year price tag of about $150 million. 
However, a more recent estimate by the city’s Independent Budget Office 
found that RSP will cost nearly $400 million.” 


New York City’s Renewal School Program 


To be classified as a renewal school, the school had to be identified as a “priority” or “focus” 
school by the New York State Department of Education; score in the bottom 25% of schools 
on state math and ELA exams; and score “proficient” or below on New York City’s most recent 
quality review.? RSP goals include: 


e “Transforming Renewal Schools into Community Schools with deepened support from and 
for families and community partners. Partnerships with community-based organizations 
will enable these schools to offer tailored whole-student supports, including mental health 
services and after-school programs. 


Creating extended learning time—an extra hour added to the school day to give all 
students additional instructional time. 


Supplying resources and supports to ensure effective school leadership and rigorous 
instruction with collaborative teachers. 


Performing school needs assessments across all six elements of the Framework for 
Great Schools (rigorous instruction, collaborative teachers, Supportive environment, 
effective school leadership, strong family-community ties, and trust) to identify key 
areas for additional resources. 


Bringing increased oversight and accountability including strict goals and clear 
consequences for schools that do not meet them.” 
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When announcing the launch of RSP, Mayor de Blasio 
pledged that, unlike Mayor Bloomberg, who “casually”5 
closed bad schools, his administration would “move 
heaven and earth” to fix failing schools.° RSP has sig- 
nificantly increased per-pupil spending at renewal 
schools, a policy that has drawn the ire of education-re- 
form advocates. (“The education industry’s cry that 
more money will solve the problem is false,”” warned 
New York governor Andrew Cuomo in 2015.) In other 
ways, however, RSP has left intact many aspects of the 
accountability policies favored by reformers, including 
lengthening the school day; setting clear, measurable 
academic goals; providing frequent feedback to school 
staff; and retaining the ability to close schools that re- 
peatedly fail to meet minimum academic standards.® 


We do know that students benefited from the previous 
administration’s accountability system, which utilized 
letter grades to boost performance and closed schools 
that failed to improve. Under Mayor Bloomberg, 
schools that received an “F” typically improved the fol- 
lowing year.? Still, some schools were closed, and dis- 
placed high school students were 15 percentage points 
more likely to graduate at their new schools than were 
similar students whose ineffective schools did not 
close.*° The new schools that emerged not only did a 
much better job educating students;" they also did so 
while spending less per pupil.” 


In this regard, Mayor de Blasio’s decision to continue 
shutting at least some bad schools that are unrespon- 
sive to turnaround efforts is welcome. Evidence from 
the previous administration demonstrates that stu- 
dents benefit greatly from escaping failing schools. 


Test Scores 


Various media reports have cited renewal schools’ low 
test scores on the spring 2016 state tests—one year after 
RSP went into full effect—as evidence that the program 
is failing.'? That is not a fair conclusion to draw. To de- 
termine the effectiveness of RSP, one must examine 
renewal schools’ relative progress: How would renewal 
schools have performed without RSP? 


The first step: Figure 1 (ELA) and Figure 2 (math) 
compare average 2013-16 test scores of renewal schools 
with average test scores of nonrenewal schools—as well 
as with average test scores for nonrenewal schools in 


FIGURE 1. 


Average ELA Scores, K-8, 2013-16 
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FIGURE 2. 


Average Math Scores, K-8, 2013-16 
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the bottom 25th and bottom 10th test-score percen- 
tiles, respectively. 


On the 2013, 2014, and 2015 tests—all taken before 
RSP went into full effect—renewal schools and nonre- 
newal schools achieved similar progress. (If anything, 
renewal schools made slightly less progress, especial- 
ly in math, than nonrenewal schools.) However, in 
2016—one year after RSP went into full effect—renewal 
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schools improved faster: the average renewal school 
improved its test score in ELA by 8.6 points and in 
math by 1.8 points, compared with 6.8 and 0.18 points, 
respectively, for the average nonrenewal school. But 
to fully understand the performance of the renewal 
schools, a more granular analysis is necessary. 


Modeling RSP’s Impact 


Methodology/Assumptions 

I use test scores and demographic information for the 
2012/13—2015/16 school years and restrict my analysis 
to grades 3-8, the only grades covered by New York 
State tests. To estimate the impact of RSP on a school’s 
test scores, I deploy a differences-in-differences model, 
which compares yearly test-score changes of renewal 
schools with those of nonrenewal schools. (The model 
assumes that—absent RSP—renewal schools would 
have remained on the same test-score trajectory.) I also 
deploy a model that compares yearly test-score changes 
of renewal schools with those of nonrenewal schools 
with comparably low test scores as of 2013. My OLS 
regression is: 


Y,,= B+ B1X,,+B, (renewal,* Post,) + 6,+ y,+ €, 


My controls are a school’s total enrollment; the pro- 
portion of its students who are female, Asian, Afri- 
can-American, Hispanic, or other (i.e., non-Caucasian); 
the proportion of its students who receive special-educa- 
tion services; the proportion who are English language 
learners; and the proportion who are in poverty. y,, is a 
school’s average test score (math or ELA) in the spring 
of the year, t. X is a school’s demographics vector. 


Post indicates that the year is after the renewal school 
designation. Because RSP was not fully operational until 
spring 2015, I run separate analyses that define Post to 
begin in either 2015 or 2016. The term renewal equals 1 
for renewal schools and o for nonrenewal schools. 6 and 
y are year and school-fixed effects, respectively. ¢ is an 
idiosyncratic error term. 32 equals the impact of RSP on 
a school’s average test scores. 


As noted, average 2016 ELA test scores rose at both 
renewal and nonrenewal schools—an outcome that was 
largely driven by a change in testing procedures.* To 
determine whether this change affected renewal schools 
differently from nonrenewal schools, I evaluate, below, 


whether my estimates are still robust when the model 
controls only for nonrenewal schools with very low test 
scores in 2013. 


During November 2014—May 2016, the number of stu- 
dents in renewal schools fell by nearly 15%.* Because 
motivated students (and thus, higher-scoring students) 
are more likely to flee bad schools, renewal schools’ 
average test scores most likely fell as a result of this 
student exodus—which, in turn, would exert a down- 
ward bias on my estimates of the magnitude of RSP’s 
impact. 


RSP’s Impact 

Figure 3 shows RSP’s impact when: (i) the renewal 
school treatment begins in 2015, i.e., before the program 
entered into full effect at all renewal schools (the 2015 
& 2016 column); and (ii) the treatment begins in 2016, 
i.e., when the policy was in full effect (the 2016 column). 
Figure 3 shows RSP’s collective impact on grades 3-8, 
as well as its impact on each grade, 3-8. 


When RSP’s impact is assessed for 2015 & 2016, it boosts 
renewal schools’ ELA scores but not their math scores. 
When RSP’s impact is assessed only for 2016, it boosts 


FIGURE 3. 


RSP’s Impact, Math and ELA 
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Source: Author’s calculations based on DOE data 


renewal schools’ average ELA and math scores: a statis- 
tically meaningful increase of 0.13 standard deviations 
(ELA) and 0.09 standard deviations (math). 


In other words, RSP boosted student achievement 
by roughly 49 days of extra schooling in ELA and 
33 days of additional schooling in math.** Under 
Mayor Bloomberg’s (discontinued) “school prog- 
ress reports,” schools that received an “F” in the 
first year of that system subsequently improved 
student scores by 36 days of extra schooling in ELA 
and 72 days of extra schooling in math.” 


As Figure 3 shows, when the treatment begins 
in 2016, the estimated impact of being a renewal 
school is positive; but it is still statistically insig- 
nificant in several grades. In both ELA and math, 
RSP’s strongest effect is seen in fourth grade. The 
pattern of estimates displayed in Figure 3 suggests 
that the results might be sensitive. Still, the results 
are consistent with an overall positive treatment 
effect that is estimated imprecisely in 

some grades. 


Figure 4 (ELA) and Figure 5 (math) 
compare yearly test-score changes of 
renewal schools with those of nonrenewal 
schools with comparably low test scores 
as of 2013 (i.e., schools with test scores 
in the bottom 25th, 10th, and 5th per- 
centiles, respectively). When the sample 


Grade 


Together, Figures 3—5 suggest that, on average, RSP 
boosted academic achievement in renewal schools. 
However, as Figure 6 shows, not all renewal schools 
are now performing up to par: indeed, 38.1% of 
renewal schools performed worse in math than 
expected—given the demographic makeup of their 
students—even with RSP. (In ELA, one-third of 
renewal schools performed worse than expected.) 


In Figure 6, dots represent renewal schools. The 
horizontal axis shows schools’ 2013 math scores. 
The vertical axis shows the difference between 
each renewal school’s 2016 math scores and the 
score that the model predicted that it should earn, 
given the demographics of its students: schools 
situated above the o line fared better than expect- 
ed; schools situated on the o line fared as expect- 
ed; and schools situated below the zero line fared 
worse than expected.’ 


FIGURE 4. 


ELA Results, Comparison of Renewal Schools with 
Low-Performing Traditional Public Schools 


Bottom 5 Percentile 
2015-16 


Bottom 25 Percentile | Bottom 10 Percentile 


2015-16 


fe restikted in ths way, Ge cotmecs (a alas 
aauean Similar but often fail to meet Grade3 | 0.0790 -0.0361 “1.334 -0.238 “1.716 “1.263 
conventional levels of statistical signifi- 

: [1.812 [2.245 [1.962 [2.386 [2.495 [2.754 
Rae This could be due to the reduced Grade 4 2.451 4.587°** 1.943 3.907 1,623 4.471" 
sample size, which makes the estimates nei) | ter) | urs) | qi4e7) 1 pes7) | 1961 
less precise. Grade 5 2.616 1714 2.679 1.494 2.472 1.632 

[1.946 [1.990 [2.017 2.111 [2.239 [2.677 

Restricting the sample to include only Grade 6 0.485 2.073 0.0436 1.900 1.017 2.437 
low-performing public schools has a larger [1.292 [1.481 [1.494 [1.736 [1.726 [2.078 
impact on the ELA estimates—a result Grade7 |  -0.990 -0.383 1.822 1.468 “1,895 -0.936 
consistent with the idea that the change in [1.103 [1.186 [1.233 [1.399 [1.521 [1.838 
testing policy could have affected scores on Grade8 | -0.371 0.961 1.209 0.509 -0.899 0.753 
the bottom of the test-score distribution [1.091 1.341 [1.292 [1.465 [1.406 [1.661 
in ELA in a way that makes estimation dif- **= significant at 5% level ** = significant at 1% level 
ficult. It also suggests that we should have 
less confidence in the ELA results than in odels are estimated via ordinary least squares. Standard errors 


the math results, which remain relatively 
consistent. (Indeed, the estimated policy 
effect increases for the math results when 
we use the more restricted sample.) 


clustered by school are reported in brackets. Each cell represents 
the results of a separate regression. 


Source: Author’s calculations based on DOE data 
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Conclusion 


Mayor de Blasio’s Renewal School Program 
aimed to fix 94 struggling New York City 
schools by promising a clean break with 
Mayor Bloomberg’s school-closure policy. 
In practice, RSP has retained many of the 
elements of the latter. Where RSP has 
notably shifted course is in its eagerness 


Grade 
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ELA; and it is statistically insignificant for 
ELA when renewal schools are compared 
only with nonrenewal schools that were 


Models are estimated via ordinary least squares. Standard errors 
clustered by school are reported in brackets. Each cell represents 
the results of a separate regression. 


Source: Author’s calculations based on DOE data 


low-performing as of 2013. 


As noted, the academic gains generated 

by RSP have not come cheaply: about $1.4 million in 
additional annual spending per renewal school. In 
contrast, Mayor Bloomberg’s school-closure policy 
produced large gains and required little, if any, addi- 
tional spending. Moreover, even when failing schools 
improve modestly, it is an open question whether the 
improvement outweighs the opportunity costs—the 
subject of a forthcoming paper by this author—which 
include turning the building space into a high-per- 
forming charter school. 


In short, these findings suggest that New York’s 
Renewal School Program has boosted student achieve- 
ment, on average, in some of the city’s worst schools; 
but RSP has done so at exceptionally high cost, espe- 
cially when compared with the low-cost, highly effec- 
tive, alternative of swiftly closing ineffective schools. 


FIGURE 6. 


Not All Renewal Schools Benefited from RSP 
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Abstract 


In November 2014, the New York City Department of Education classified 
94 low-performing schools as “renewal schools.” New York’s Renewal School 
Program (RSP), announced Mayor Bill de Blasio at the program’s launch, 
represented a big change to the city’s education policy: whereas former 
mayor Michael Bloomberg favored closing failing schools, RSP would fix 
them instead. 


RSP—which went into full effect in the spring of 2015—has retained many of 
the core elements of a strong accountability policy, including school oversight 
and more time in the classroom (renewal schools have longer days). RSP 
supplements these stalwarts of education reform with an infusion of extra 
money: $149 million over three years. 


Is RSP succeeding? The release of renewal schools’ spring 2016 state test 
results in math and English language arts (ELA) allows for a one-year 
evaluation. This report applies regression models to measure how renewal 
schools perform relative to how they would have fared absent RSP. 

Key Findings 


1. RSP has boosted student achievement by a meaningful magnitude, 
on average. 


2. When the demographic profiles of their students are accounted for, 
about 33.3% of renewal schools still perform worse than expected in 
ELA, and 38.1% perform worse than expected in math. 


3. RSP is also significantly more costly than Mayor Bloomberg’s 
school-closure policy. 
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